In Augmented Reality (AR), a view of a real environment, such as a video image of the real environment, is combined with an overlay of one or more virtual objects in a spatial relationship to the real environment. For many AR applications, the virtual objects shall seamlessly integrate in the view so that real and virtual objects cannot be distinguished. Therefore, it is important to illuminate or display the virtual objects with the same lighting conditions visible in the real world as well as let the virtual objects change the illumination for example by casting shadows onto parts from the real scene. For augmented reality scenes, the lighting conditions are typically unknown and arbitrary, and therefore it is generally difficult or even impossible to have consistent lighting for the real and virtual objects.
A possible way to have consistent lighting for the real and virtual objects in AR applications is to estimate light emitted from the real environment. Common approaches in state of the art require additional setups, e.g. mirrors, or special cameras, e.g. fish eye camera, in order to estimate environment light. This definitely restricts applicability of these approaches. Further, most of the common approaches could only estimate directions of environment light, but not positions of any light source.
An approach of estimating or capturing environment light is to create a panoramic image of the real environment by an omni-camera (like described in Sato, I., et al., “Acquiring a radiance distribution to superimpose virtual objects onto a real scene,” IEEE Transactions on Visualization and Computer Graphics 5.1 (January-March 1999) (“Sato”)) or by capturing an image of a mirror sphere (like described in Debevec). The environment light could be directly measured according to the panoramic image. One problem of this approach is the necessity of additional setup, which, e.g., either requires a fish-eye camera lens or the mirror sphere, wherein the extra objects added to the scene change the original scene. Further, this approach measures directions of the environment light.
Another approach of estimating environment light is based on an image of the real environment and a known model of the real environment. For example, in Gruber, Lukas, et al., “Real-time photometric registration from arbitrary geometry,” IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2012 (“Gruber”) propose to use a RGB-D camera to capture a depth image of the real environment and then reconstruct a surface of the real environment. The environment light could be estimated from the surface information based on the depth data and texture information from the RGB data from the RGB-D camera.
Therefore, it would be desirable to provide a method of representing a virtual object in a view of a real environment which is capable of enhancing applicability of an augmented reality application, particularly in environments with unknown lighting conditions.